ARock: an Algorithmic Framework for Async-Parallel Coordinate Updates

The problem of finding a fixed point to a nonexpansive operator is an abstraction of many models in numerical linear algebra, optimization, and other areas of scientific computing. To solve this problem, we propose ARock, an asynchronous parallel algorithmic framework, in which a set of agents (machines, processors, or cores) update randomly selected coordinates of the unknown variable in an asynchronous parallel fashion. The resulting algorithms are not affected by load imbalance. When the coordinate updates are atomic, the algorithms are free of memory locks. We show that if the nonexpansive operator has a fixed point, then with probability one, the sequence of points generated by ARock converges to a fixed point of the operator. Stronger convergence properties such as linear convergence are obtained under stronger conditions. As special cases of ARock, novel algorithms for linear systems, convex optimization, machine learning, distributed and decentralized optimization are introduced with provable convergence. Very promising numerical performance of ARock has been observed. Considering the paper length, we present the numerical results of solving linear equations and sparse logistic regression problems.